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(54) SIGNAL PROCESSING DEVICE AND METHOD THEREFOR AND PROGRAM STORING 
MEDIUM 



(57) In a signal processing device and its method as 
well as a program storing medium, psychological audi- 
tory sense analysis of the survival state of an input signal 
of the time when the input signal has been compressed 
is perfomned in response to the compression process- 
ing, and the result of the very analysis is outputted as 
the psychological auditory sense encoded Infonnation, 



and digital watemiarking infonnation is superimposed 
on the Input signal on the basis of the psychological au- 
ditory sense encoded information, so that it is possible 
to easily realize digital watermarking Information that 
has large compression resistance and digital water- 
marking information that has small compression resist- 
ance. 
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Description 

Teclinical Field 

[0001] This invention is directed to a signal process- 
ing device and its method as well as a program storing 
medium, and suitably applicable to the safeguarding of 
the copyright of music data and so on. 

Background Art 

[0002] In recent years, music contents and so on have 
been distributed to an indefinite number of networlc ter- 
minal devices (computers) from a Web site via a network 
such as the Internet. 

[0003] A computer user is able to record the distribut- 
ed music contents on a wide variety of recording media 
In a state of digital data, so as to generate a music soft 
of favorite music that is recorded with a satisfactory tone 
quality, as the user wishes. 

[0004] In this way, a computer user has accessed a 
Web site and recorded music contents on a recording 
medium freely. However, the act of distributing music 
contents by the person who set up the Web site without 
permission of the copyrighter, who is the owner of the 
music contents, will cause a problem of an infringement 
of the copyright of the music contents. 

Disclosure of Invention 

[0005] Considering aforementioned respects, this in- 
vention proposes a signal processing devtee and its 
method as well as a program storing medium that can 
effectively suppress the infringement of the copyright of 
contents that have been distributed to an indefinite 
number of temninal devices via, for instance, a network 
[0006] In order to solve such a problem, in the present 
invention, psychological auditory sense analysis of the 
survival state of an Input signal of the time when the In- 
put signal has been compressed is performed in re- 
sponse to the compression processing, and the result 
of the very analysis is outputted as the 
psychological auditory sense encoded Infomriation, and 
digital watemnarklng infonmation is superimposed on the 
input signal on the basis of the psychological auditory 
sense encoded information, so that it is possible to eas- 
ily generate digital watenmaricing infomriation that has 
large compression resistance and digital watennaricing 
infomiation that has small compression resistance. 

Brief Description of Drawings 

[0007] 

Fig. 1 Is a block diagram illustrating a general con- 
figuration of a copy control system in which a wa- 
temnaric is utilized according to the present inven- 
tion. 



Fig. 2 Is a schematic diagram showing an example 
of application of the copy controlling in the copy con- 
trol system. 

Fig. 3 is a block diagram illustrating a configuration 
s of the watemiaric encoder according to an embodi- 
ment of the present Invention. 
Fig. 4 is a schematic diagram used for explaining 
the principle of superimposing a watennark. 
Fig. 5 is a schematic diagram used for explaining 
10 the processing of the psychological auditory sense 
analysis division. 

Fig. 6 Is a schematic diagram used for explaining 
the processing of the psychological auditory sense 
analysis division. 

Fig. 7 is a schematic diagram used for explaining 
the psychological auditory sense encoding. 
Fig. 8 is a schematic diagram showing an example 
of encoding of the watermarks according to the 
present invention. 

Fig. 9 is a flow chart showing a procedure of the 
watennark encoder 

Fig. 1 0 is a block diagram Illustrating a configuration 
of the watennark decoder. 
Fig. 11 is a flow chart showing a procedure of the 
watennark decoder. 

Fig. 12 Is a block diagram illustrating an internal 
configuration of a computer. 

Best Mode for Carrying out the Invention 

[0008] An embodiment of the present invention will be 
described in detail hereinafter with reference to the 
drawings. 

[0009] tn Fig. 1, 10 shows a contents copy control sys- 
tem as a whol e ; at the time wh en it records a digital audio 
signal DA, which is the audio source, on an optical disk 
2, it adds a robust (strong) watermaric DC1 and a fragile 
(weak) watennark DC2 to the digital audio signal DA and 
then records them on the optical disk 2, by means of a 
watennark (digital watennaridng information) encoder 
1. 

[0010] At here, the robust watennark DC1 is com- 
prised of information of the audio source, such as the 
name of the copyright holder, and whether the copying 
is pemnltted or not; the fragile watennaric DC2 is com- 
prised of infomriation such as a trigger for detecting the 
compression history. In this connection, the robust wa- 
termari^ DC1 Is a watennark that does not vanish even 
if it has passed through a compressing system of MP3 
and so on, and the fragile watennark DC2 is a water- 
mark that vanishes (the survival rate becomes low) 
when it has passed through a compressing system of 
MPS and so on, because of its quantization error 
[0011] In this way, in the case where a compressed 
stream signal DS, which has been uploaded to a site on 
a network 4 via a compression processing by, for in- 
stance, a MPS (MPEG Audio Layer S) encoder S and so 
on, out of optk:ai disks 2 on which the digital audio signal 
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DA including the robust watermark DC1 and the fragiie 
watemiark DC2 has been recorded, is downloaded to a 
personal computer 5, and also in the case where an op- 
tical disk 2 that is distributed without being compressed, 
out of optical disks 2 on which the digital audio signal 
DA Including the robust watermark DC1 and the fragile 
watennark DC2 has been recorded, is loaded to the per- 
sonal computer 5, the personal computer 5 first judges 
based on a signal that Is input to a Codec (Code and 
Decode) comprising a modem (Modulator Demodula- 
tor) whether the input signal to the personal computer 5 
is the non-compressed digital audio signal DA or the 
compressed stream signal DS, and by what compres- 
sion standard the compressed stream signal DS has 
been encoded and then selectively switches a switch 
SW according to the judgement result. 
[0012] And, the personal computer 5 detects the em- 
bedded copyright information out of the input signal of 
the personal computer 5 by means of a watennark de- 
coder 6, and controls copying of the Input signal (the 
digital audio signal DA, the compressed stream signal 
DS and so on) to a PD (portable device) 8 such as a 
memory card. That is, the watermark decoder 6 detects 
the robust watemiark DC1 and the fragile watermark 
DC2. A LCM (licensed SDMI (secure digital music Initi- 
ative) compliant module) 7 controls whether the music 
contents are copied to the PD 8 or not, on the basis of 
two kinds of watennarks (the robust watennark DC 1 and 
the fragile watennark DC2) that have been detected by 
the watennark decoder 6. 

[0013] Fig. 2 shows an example of controlling of 
whether the music contents are copied to the PD 8 or 

not, which is perfomried by the LCM 7, using the detect- 
ed two kinds of watennarks; in the case where the robust 
watennark DC1 has been detected and the fragile wa- 
tennark DC2 has not been detected, the LCM 7 deter- 
mines that the musk: contents have some compression 
histories, and inhibits the PD B from copying the very 
music contents. 

[0014] In the case where both the robust watermark 
DC1 and the fragile watennark DC2 are detected, the 
LCM 7 determines that no compression history could be 
found and pennits copying of the very music contents 
to the PD 8. 

[0015] Furthennore, in thecasewhereboth the robust 
watermark DC1 and the fragile watermark DC2 are not 
detected, the LCM 7 pennits copying of the very music 
contents to the PD 8 because of no information as to the 
copyrighter of the music content or pennission of copy- 
ing. There is no technical possibility of a case where the 
robust watermark DC1 is not detected and the fragile 
watermark DC2 is either detected, which is thus excep- 
tional. 

[001 6] Fig. 3 shows a configuration of the watermark 
encoder 1 ; a psychological auditory sense analysis di- 
vision 1 1 performs the analysis of the sequentially input- 
ted digital audio signal DA1 , utilizing a psychological au- 
ditory sense model, and controls a watennark superim- 



posing division 16 that has a robust watennark super- 
imposing division and a fragile watermark superimpos- 
ing division. The digital audio signal DA1 generates a 
MDCT coefficient D1 4 at a MDCT (modified discrete co- 
5 sine transfonn) processing division 1 4, and then outputs 
this to the watennark superimposing process division 
16. 

[0017] The watennark superimposing division 16 is 
adapted to respectively superimposing the robust wa- 
10 termaric DC1 and the fragile watermark DC2 on a spec- 
trum of the frequency band that has been detemnined 
on the basis of the analysis result of the psychological 
auditory sense analysis division 11 mentioned below, 
out of a frequency spectrum of time series sample 
IS blocks of the digital audio signal DA1 that has been MD- 
CT converted. 

[0018] More concretely, the watenmaric superimpos- 
ing division 16 extracts a spectrum of the frequency 
band f1 on which to superimpose the robust watermaric 
DC1 and the fragile watermark DC2, out of the frequen- 
cy spectrum of time series sample blocks of the input 
audio signal DA1 that has been MDCT converted (Fig. 
4(A)). 

[001 9] The watermaric superimposing division 1 6 then 
scales the frequency spectrum f 1 at a stated rate, and 
generates a frequency spectrum f2 that is attenuated 
(Fig. 4(B)). If the polarity of data to be embedded is "1", 
the frequency spectrum f2 is used as it is. If the polarity 
of data to be embedded is "0", the frequency spectrum 
f2 with the positive and negative inverted is used. 
[0020] The watennaric superimposing division 16 
shifts the frequency spectrum f2 In the direction of higher 
or lower frequency to keep a distance for four bars of 
spectrum (Fig. 4(C)). In Fig. 4C, the frequency spectrum 
f2 is shifted for four bars of spectrum In the direction of 
higher frequency to generate a frequency spectrum f3. 
[0021] Next, the watermark superimposing division 
16 superimposes the robust watennark DC1 and the 
fragile watennari< DC2 on a frequency spectrum f4 (Fig. 
4 (D)), which has been obtained by adding the shifted 
frequency spectrum fS (Fig. 4 (C)) to the original fre- 
quency spectrum f1 (Fig. 4(A)). 
[0022] At the point of addition, the con-elation between 
the polarities of a certain frequency spectrum and a fre- 
quency spectrum that is away by four bars is processed 
to be inclined to the positive or negative. This processing 
is performed only when the sign of the frequency spec- 
trum f4 is inverted by adding the frequency spectrum f3 
to the frequency spectrum f 1 , but not perfomned at other 
times when the sign is not inverted. In Fig. 4(D), blank 
frames represent signal elements eliminated from the 
frequency spectrum f1 (Fig. 4(A)) and hatching frames 
represent signal elements obtained from the addition. 
[0023] This can reduce the amount of embedded wa- 
termari<s which causes deterioration of tone quality and 
does not contribute to the accuracy of detection. More- 
over, by scaling the original frequency spectrum and 
adding it to its adjacency, embedding with reduced de- 
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terioration of tone quality will be possible using the 
mas Icing effect. 

[0024] In this connection, if it adds the level that has 
been obtained by attenuating the level of the first fre- 
quency component f 1 (Fig. 4(A)) In which the watemiark 
Is to be embedded in a definite rate and that the polarity 
of the watermark to be embedded is multiplied, to the 
level of the second frequency component f3 (Fig. 4(C)) 
that is away from the first frequency component f2 (Fig. 
4(B)) by, for example, 4 frequency components, and em- 
beds the watemfiark, then, after the watemnark has been 
embedded, such a probability is heightened that the po- 
larity of the correlation between the first frequency com- 
ponent f1 (Fig. 4(A)) and the level of the second frequen- 
cy component f3 (Fig. 4(C)) becomes the same (positive 
or negative), and this is utilized for detection of the wa- 
termark in a watennark decoder that will be mentioned 
below. 

[0025] In this way, on the basis of the result D11 ofthe 
analysis perfomned in the psychological auditory sense 
analysis division 11 , the watemnark superimposing proc- 
ess division 16 embeds the robust watermark DC1 and 
the fragile watemnark DC2, so as to become the fre- 
quency band and the sound pressure level that include 
the consideration for the threshold of audibility and/or 
the masking threshold effect mentioned below, in the 
previously and separately allocated plural frequency 
bands, respectively. 

[0026] That Is, the psychological auditory sense anal- 
ysis division 1 1 analyzes the input digital audio signal 
DA1» and calculates from curves of original sound and 
threshold of audibility a masking threshold which Is the 
limit to being able to perceive actually listening to the 
original sound. The masking capability toward the given 
digital audio signal DAI depends on the frequency band 
and the level of sound pressure. 
[0027] Generally speaking, in an encoder for com- 
pression processing, on the basis of the infomnation of 
the frequency band and the level of sound pressure, the 
best method for representing the input audio signal with 
the restricted bit resolution would be determined. That 
is, by utilizing a psychological auditory sense model that 
is used for compression, and embedding the watermark 
with a position and an energy of the frequency that tends 
to be eliminated by the compression process, It is pos- 
sible to realize the fragile watermark DC2. 
[0028] Hereinafter, the principle of the method for em- 
bedding the robust watemnark DC1, which is resistant 
to compression, and the fragile watemnark DC2, which 
Is easily affected by compression, utilizing the frequency 
band that tends to be eliminated by compression proc- 
ess is explained. 

[0029] Fig. 5 shows a threshold of audibility; a thresh- 
old of audibility of silent time means the minimum level 
of sound that can be detected by the auditory sense, 
and relates to a boundary of a noise that can be per- 
ceived by the auditory sense at a silent time. As shown 
in Fig. 5, a sound whose sound pressure level is higher 



than the threshold of audibility of silent time like the tone 
A can be perceived, but a sound whose sound pressure 
level Is lowerthan the threshold of audibility of silent time 
like the tone B can not be perceived. Moreover, as 
s shown in Fig. 5, a threshold of audibility of silent time 
depends on frequency, and even if the tones A and B 
have the same sound pressure level, they can be per- 
ceived or not depending upon the frequency of the 
sound. 

10 [0030] And, Fig. 6 shows a masking effect; a masking 
effect mainly relates to the detection threshold of quan- 
tization distortion and/or background noise, and the de- 
tection threshold of the specific sound greatly varies de- 
pending on the other sounds that are heard at the same 

IS time. As shown in Fig, 6, as to a definite frequency range 
concerning the tone C, the other sounds become difficult 
to be perceived; for instance, the tone D is almost inau- 
dible in spite of the fact that it is a pure tone that has 
comparatively high sound pressure, on the other hand, 

20 the tone E is audible. The nearer the frequency of the 
masker (a sound that masks the other) and the frequen- 
cy of the maskee (a sound that is masked and made 
inaudible) are positioned, the stronger the masking ef- 
fect operates. 

25 [0031] Therefore, in the watemnark encoder 1 , utiliz- 
ing these principles, the process of embedding the wa- 
temnarks (the robust watennark DC1 and the fragile wa- 
termark DC2) is achieved wherein the compression 
characteristics of MP3 is considered. 

30 [0032] In this connection, Fig. 7 shows encoding (psy- 
chological auditory sense encoding in which psycholog- 
\ca\ auditory sense Is considered) perfomned in a MPS 
encoders (Fig. 1); for instance, after the MPS encoder 
3 has divided the input audio signal DA into 32 frequency 

35 bands (subbands), it perfomns psychological auditory 
sense encoding in such a manner that the quantization 
noise falls under the threshold of audibility. The solid line 
in Fig. 7 represents a frequency distribution ofthe audio 
signal OA, and the bold line represents the threshold of 

40 audibility of silent time. 

[0033] In psychological auditory sense encoding, in 
the first place, from the curves of the original sound and 
the threshold of audibility, the masking threshold (the 
broken line in Fig. 7) that Is the boundary which can be 

45 perceived during the original sound is actually heard is 
calculated. Next, for each subband, the quantization 
step of each subband is allocated, in such a manner that 
the quantization noise becomes the smaller level than 
the masking threshold. 

50 [0034] In Fig. 7, the ranges that are sampled by the 
compression processing are shown with the rectangle 
areas, and the bases of the rectangles correspond to 
the quantization noise levels. In such a region that the 
original sound is smaller than the masking threshold, the 

55 original sound is inaudible, and so it can be eliminated 
from the sample; on the other hand, in such a region 
that the sensitivity of auditory sense is high and the 
masking do not work sufficiently, the quantization steps 
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are made fine, and the level of the quantization noise is 
lowered hereby. 

[0035] Therefore, the psychological auditory sense 
analysts division 11 of the watennark encoder 1 analyz- 
es the digital audio signal DAI based on the character- 
istics of psychological auditory sense encoding of the 
MP3 encoder 3 with respect to the inputted digital audio 
signal DA1, and, on the basis of the result of the very 
analysis, calculates the frequency band and its sound 
pressure in which the robust watemnark DC1 and the 
fragile watennark DC2 are to be ennbedded, and con- 
trols the watenmark superimposing division 1 6 by the re- 
sult of the very calculation (the psychological auditory 
sense encoded information D11). 
[0036] As a result of this, as shown in Fig. 8, by em- 
bedding a watermark in a rectangular region, which is 
sampled In compression processing, the very water- 
mark would function as the robust watennark DC1 that 
is resistant to compression, and if a watennark has been 
embedded in the outside of a rectangular region, which 
is eliminated through compression processing, the very 
watennark would function as the fragile watennark DC2 
that is easily affected by compression. 
[0037] On the basis of the psychological auditory 
sense encoded infomnation D11 that has been calculat- 
ed by the psychologteal auditory sense analysis division 
11, the watermark superimposing division 16 embeds 
the robust watermark DC1 and the fragile watermark 
DC2 in the digital audio signal DAI . 
[0038] In this connection, the region in which the wa- 
tennark is embedded is not limited to a frequency re- 
gion, but may be a time region. 
[0039] Then, a MDCT coefficient D16, which is out- 
putted from the watennark superimposing division 16, 
is exposed to inverse orthogonal transfonnation in an 
IMDCT processing division 1 5, in this way it is outputted 
as the digital audio signal DA2 that is having the same 
form as the form of the original digital audio signal DA1 
and that the robust watennark DC1 and the fragile wa- 
tennark DC2 have been embedded in. 
[0040] In this embodiment, the digital audio signal 
DA2 is encoded by the stated encoder (not shown), and 
used to expose an original master of a disk, and then 
an optical disk 2 is produced in volume on the basis of 
this original master disk. 

[0041 ] Fig. 9 shows a procedure of the watermark en- 
coder 1 , which processes the digital audio signal DAI ; 
the watermark encoder 1 moves from the step SP11 to 
the step SP12, and sequentially reads the digital audio 
signal DA1 for each sample block. Then, the watermark 
encoder 1 goes to the step SP1 3, and analyzes the dig- 
ital audio signal DAI , and extracts the psychological au- 
ditory sense encoded information D11 from the digital 
audio signal DAI . 

[0042] Then, on the following step SP14, the water- 
marlc encoder 1 embeds the robust watennark DC1 in 
the digital audio signal DA1 , on the basis of the psycho- 
logical auditory sense encoded infomiation D11 ; in ad- 



dition, on the step S PI 5, it embeds the fragile watennark 
DC2 in the digital audio signal DAI , on the basis of the 
psychological auditory sense encoded infomnation D11 . 
[0043] In this way, the watennark encoder 1 obtains 
5 the digital audio signal DA2 that is having the same form 
as the form of the inputted digital audio signal DA1 and 
that the robust watermark DC1 and the fragile water- 
mark DC2 have been embedded in. 
[0044] The watemnark encoder 1 then goes to the step 
10 SP1 6, and judges whether the processing of the digital 
audio signal DAI has been completed or not; when the 
negative result has been obtained here, it returns to the 
stepSP12. In this way, the watermark encoder 1 repeats 
this procedure for each sample block in sequence, so 
^5 as to process the digital audio signal DAI ; when the af- 
finnative result has been obtained on the step SP16, it 
moves from the step SP1 6 to the step SP1 7, and ends 
the very procedure. 

[0045] Fig. 1 0 shows a configuration of the watennark 
decoder 6 of the personal-computer 5, which has been 
stated above with reference to Fig. 1 ; the watennark de- 
coder 6 detects the robust watennark DC1 and the frag- 
ile watemnark DC2 from the digital audio signal DA2, 
which Is obtained by a playback of the optical disk 2 (Fig. 
1). 

[0046] That is, the watennark decoder 6 perfonns 
MDCT processing of the sequentially inputted digital au- 
dio signal DA2 in a MDCT processing division 60, and 
obtains a MDCT coefficient D60 hereby, and then out- 
puts this to the watermark detecting division 61 . 
[0047] With respect to the Inputted MDCT coefficient 
D60, the watennark detecting division 61 shifts each of 
plural frequency components that have been previously 
and separately allocated to the robust watennark DC1 
and the fragile watenmark DC2 as their embedding fre- 
quency bands by, for instance, 4 frequency components 
in such a direction that the frequencies Increase; on the 
basis of the new MDCT coefficient that has been ob- 
tained by the very shifting process and the MDCT coef- 
ficient D60 that is obtained from the MDCT processing 
division 60, it compares the polarities of the frequency 
components, and detects the robust watennark DC1 
and the fragile watennark DC2 on the basis the lean of 
the polarity. 

[0048] In this connection, the region in which the wa- 
termark is detected is not limited to a frequency region, 
but may be a time region. 

[0049] When only the robust watennark DC1 has 
been detected at the time of copying of the digital audio 
signal DA2 that has been recorded on the optical disk 
2, this is representing that there is a history of compres- 
sion based on MP3, that is, the digital audio signal DA2 
In which the robust watennark DC1 that has been de- 
tected at this time has been embedded is the contents 
that has been distributed via the network 4; so, when 
copying restriction is previously set by the copyrighter, 
by restricting the copying, it is possible to prevent illegal 
copying that constitutes an infringement of the copy- 
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right. 

[0050] Besides, when both of the robust watermark 
DC1 and the fragile watermark DC2 have been detect- 
ed, this is representing that there is not any history of 
compression based on MPS, that is, it is not what has 
been distributed via the network 4, and that the copy- 
righter releases it free of charge; so. it can be known 
that the copying restriction is needless. 
[0051] Thus, in the personal computer 5, it is possible 
to restrict the copying, on the basis of the copyright in- 
fomiation and the compression history information 
based on the robust watermark DC1 and the fragile wa- 
tennark DC2. 

[0052] Fig. 1 1 shows procedure of the watennark de- 
coder 6 for processing the digital audio signal DA2; the 
watermark decoder 6 enters the very procedure at the 
step SP21 and goes to the step SP22, and reads the 
digital audio signal DA2 in sequence for every sample 
block. Then, the watennark decoder 6 goes to the step 
SP23 and detects the robust watermark DC1 from the 
digital audio signal DA2, and detects the fragile water- 
mark DC2 at the following step S P24. 
[0053] After the detection of the fragile watermark 
DC2, the watermark decoder 6 goes to the step SP25 
and judges whether the processing of the digital audio 
signal 0A2 has been completed or not; when the nega- 
tive result has been obtained, itretumstothestepSP22. 
In this way, the watermark decoder 6 repeats this pro- 
cedure in sequence for each sample block, so as to 
process the digital audio signal DA2; when the affinma- 
tfve result has been obtained at the step SP25, it goes 
to the step SP26 and ends the very procedure. 
[0054] In the above configuration, the psychological 
auditory sense analysis division 11 of the watermark en- 
coder 1 determines the embedding position (for in- 
stance, frequency band) and the sound pressure level, 
in response to the compression characteristics of the 
MP3 encoder 3 (psychological auditory sense models 
of threshold of audibility, masking effect and so on that 
have been adopted in the various compression stand- 
ards), so that it is possible to embed the fragile water- 
mark DC2 that is easily affected by compression and 
that meets the compression characteristics of the MP3 
encoder 3 in the digital audio signal DAI . 
[0055] Therefore, when the fragile watemnari< DC2 
has been compressed via the IVIP3 encoder 3, its sur- 
vival rate would be lowered sufficiently, and it would not 
be detected at the watermark decoder 6. 
[0056] In this way, according to the above configura- 
tion, it Is possible to easily realize the robust watennark 
DC1 that is resistant to compression and the fragile wa- 
temnark DC2 that is affected by compression. In this 
connection, by embedding copyright infomnation In the 
robust watermark DC1 , which has strong compression 
resistance, it is possible to render the copyright infomna- 
tion difficult to be detected on the basis of the robust 
watenrnaric DC1 that has been detected after it has 
passed through the compression system. Moreover, the 



compression history can be determined on the basis of 
the reduction or elimination of the fragile watermari( DC2 
after It has passed through the compression system so 
that the illegal copying on the network can be prevented. 
5 [0057] In the above embodiment, such a case has 
been described that the robust watermark DC1 and the 
fragile watermark DC2 are detected from the digital au- 
dio signal that is obtained by a playback of the optical 
disk 2; however, we do not intend to limit the present 
invention to such. The present invention is applicable to 
also such a case that the watermark is detected from a 
stream signal in which the digital audio signal has been 
compress -processed. If do it like this, it is possible to 
eliminate trouble of extending the compressed stream 
signal and transfonning it to the same form as that of 
the original digital audio signal, and it is possible to en- 
hance the speed of the processing. 
[0058] When embedding the robust watemriaric DC1 
and the fragile watennark DC2 in the digital audio signal , 
the fragile watennaric DC2 may be embedded in a form 
that it may not be deleted by predetemnined compres- 
sion processing. 

[0059] In this case, the fragile watennark DC2 may be 
embedded in a fomri that it may not be deleted by proper 
compression processing performed at the time of distri- 
bution, and also embedded in a fomn that it may be de- 
leted when the digital audio signal with a watennark is 
compressed by other types of compression processing 
such as MP3. More concretely, compression character- 
istic of the proper compression processing used at the 
time of distribution and that of other types of compres- 
sion processing are compared. The proper compression 
processing used at the time of distribution does not de- 
lete the fragile watennark DC2. The other types of com- 
pression processing embeds the fragile watennark DC2 
into signals that are deleted by quantization processing 
for the digital audio signal, based on the above-men- 
tioned threshold of audibility and masking threshold at 
the time of compression processing. 
[0060] Furthermore, in the above embodiment, such 
a case has been described that the robust watermaric 
DC1 and the fragile watermark DC2 are embedded in 
the digital audio signal; however, we do not intend to 
limit the present invention to such. The robust water- 
mark DC1 is embedded in the digital audio signal, and 
after the compression processing of the signal, the frag- 
ile watermaric DC2 may be embedded in a compressed 
stream signal. As a result. It is possible to easily set the 
deletion level of the fragile watermark DC2 by compres- 
sion processing to the minimum, regardless of the char- 
acteristic of the Codec. 

[0061] Moreover, in the above embodiment, such a 
case has been described that the fragile watemriaric DC2 
is embedded in the digital audio signal at the position 
and level to be completely deleted after compression 
processing such as MP3; however, we do not intend to 
limit the present invention to such. The fragile water- 
mark DC2 may not be completely deleted even after the 
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compression processing such as MPS. 
[0062] The detection division detemriines whether or 
not the detected level of the fragile watemiark DC2 is 
lower than a predetemriined threshold. If the level is be- 
low the threshold, it may be set to determine that the 
digital audio signal to be detected has been illegally cop- 
ied. If beyond the threshold, it may be set to determine 
that the digital audio signal to be detected has been 
property copied. After all, the fragile watenmaric DC2 
may be embedded in the digital audio signal in any com- 
position, as long as the detection division can detect the 
fact that the compression processing has been per- 
formed according to how low the survival rate of the frag- 
ile watermark DC2 has become after the digital audio 
signal embedded with the robust watermark DC1 and 
the fragile watermark DC2 is compressed such as by 
MP3. 

[0063] Besides, in the above embodiment, such a 
.case has been described that the robust watermark DC 1 
and the fragile watermark DC2 are embedded in the dig- 
ital audio signal, using the psychological auditory sense 
model that is the compression characteristics of the 
MP3; however, we do not Intend to limit the present In- 
vention to such. The present invention can be widely ap- 
plied to such cases that psychological auditory sense 
models based on a wide variety of compression stand- 
ards such as I^PEG-AAC, ATRAC, ATRAC2, ATRAC3, 
Dolby ACS, MS (Microsoft) Audio, Twin VQ, or combi- 
nations of these psychological auditory sense models 
are used. If do it like this, it is possible to realize a gen- 
eral-use fragile watermark that suits much compression 
standards. 

[0064] Besides, in the above embodiment, such a 
configuration has been described that the robust water- 
mark encoding division has been integrated with the 
fragile watemnark encoding division in the watemnark 
encoder 1 ; however, we do not intend to limit the present 
invention to such. For instance, the present invention is 
able to use such a configuration that the robust water- 
mark encoding division is connected in series with the 
fragile watemnark encoding division, or such a configu- 
ration that the robust watenmark encoding division is 
connected in a parallel arrangement with the fragile wa- 
tennark encoding division. 

[0065] Besides, in the above embodiment, such a 
configuration has been described that the robust water- 
mark decoding division has been integrated with the 
fragile watennark decoding division in the watermark 
decoder 6; however, we do not intend to limit the present 
invention to such. For instance, the present invention is 
able to use such a configuration that the robust water- 
mark decoding division is connected in series with the 
fragile watermark decoding division, or such a configu- 
ration that the robust watennark decoding division is 
connected in a parallel arrangement with the fragile wa- 
tenmark decoding division. 

[0066] Besides, in the above embodiment, such a 
case has been described that the copyright information 



and the compression history infonnation is superim- 
posed on the digital audio signal; however, we do not 
intend to limit the present invention to such. The present 
invention may be applied to the cases where the various 
5 infonnation is superimposed on, as necessary, and 
transferred. 

[0067] Besides, in the above embodiment, such a 
case has been described that the embedding process 
of the watermark is performed by the watennark encod- 

10 er 1 ; however, we do not intend to limit the present in- 
vention to such. Alternatively, from a program storing 
medium (a floppy disk, an optical disk and so on) in 
which a program for realizing the respective functions 
of the water mark encoder 1 has been stored, the pro- 

15 gram may be loaded in an information processing de- 
vice (computer), and the respective functions may be 
executed. 

[0068] For example, an ordinary computer 70 as 
shown in Fig. 12 comprises a CPU (Central Processing 

20 Unit) 71 in charge of an entire operation, a ROM (Read 
Only Memory) 72 storing various kinds of sofhware, a 
RAM (Random Access Memory) 73 as a work memory 
for the CPU 71 , a hard disk drive 74 storing various data, 
and a communication port 75 which is an interface for 

25 communicating with the externals via a network. These 
are interconnected though a bus 76 to make a configu- 
ration. 

[0069] The hard disk drive 74 or the ROM 72 stores 
programs to perfonn each function mentioned above in 

30 the computer 70. At the time of execution, the CPU 71 
may activate each corresponding function part accord- 
ing to the work memory stored in the RAM 73. Also, the 
computer 70 may Install programs for activating each 
function mentioned above via the communication port 

35 75. 

Industrial Applicability 

[0070] This invention is directed to a signal process- 
40 ing device and its method as well as a program storing 
medium, and applicable to the safeguarding of the cop- 
yright of music data and so on. 



1 . A signal processing device for superimposing digit- 
al watermaricing Information on the given input sig- 
nal, said device comprises: 

50 

psychological auditory sense analysis means 
for pertonning psychological auditory sense 
analysis of the survival state of said input signal 
of the time when said input signal has been 
55 compressed, in response to said compression 

processing, and for outputting the result of the 
very analysis as the psychological auditory 
sense encoded information; and 



55 



45 Cia ms 



7 



13 



EP 1 202 250 A1 



14 



superimposing means for superimposing said 
digital watemnarking infomnation on said input 
signal, on the basis of said psychological audi- 
tory sense encoded infomnation. 

5 

2. The signal processing device according to claim 1 , 

wherein 

said psychological auditory sense analysis 
means generates said psychological auditory 
sense encoded infomnation, on the basis of the io 
threshold of audibility. 

3. The signal processing device according to claim 1 , 
wherein 

said psychological auditory sense analysis i5 
means generates said psychological auditory 
sense encoded infomnation, on the basis of the 
masking effect of the audio signal. 



8. The signal processing method according to claim 6, 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- 
fomnation is generated, on the basis of the masking 
effect of the audio signal. 

9. The signal processing method according to claim 6, 
wherein 

at said step of superimposing said digital wa- 
termarking infonnation on said Input signal, digital 
watemnarking information that has large compres- 
sion resistance toward said compression process- 
ing and digital watermarking information that has 
small compression resistance toward said com- 
pression processing are supeririiposed on said in- 
put signal, on the basis of said psychological audi- 
tory sense encoded information. 



4. The signal processing device according to claim 1 , 
wherein 

said superimposing means superimpose dig- 
ital watermari<ing infonnation that has large com- 
pression resistance toward said compression 
processing and digital watennarking information 
that has small compression resistance toward said 
compression processing on said input signal, on the 
basis of said psychological auditory sense encoded 
infomnation. 

5. The signal processing device according to claim 1 , 
wherein 

said digital watemriaricing Information Is the 
copyright Information relevant to said Input signal. 

6. A signal processing method of superimposing dig- 
ital watenmarking information on the given input sig- 
nal, said method comprises: 

a step of perfonning psychological auditory 
sense analysis of the survival state of said input 
signal of the time when said input signal has 
been compressed, in response to said com- 
pression processing, and then outputting the 
result of the very analysis as the psychological 
auditory sense encoded infonrkation; and 
a step of superimposing said digital watemnark- 
ing infomnation on said Input signal, on the ba- 
sis of said psychological auditory sense encod- 
ed information. 

7. The signal processing rnethod according to claim 6, 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- 
formation is generated, on the basis of the threshold 
of audibility. 



20 1 0. The signal processing method according to claim 6, 
wherein 

said digital watemnarking Infomnation is the 
copyright information relevant to said input signal. 

25 11. A program storing medium for causing a signal 
processing device to execute a program compris- 
ing: 

a step of perfomning psychological auditory 
30 sense analysis of the survival state of said Input 

signal of the time when said input signal has 
been compressed, in response to said com- 
pression processing, and then outputting the 
result of the very analysis as the psychological 
35 auditory sense encoded infonnation; and 

a step of superimposing said digital watemnark- 
ing information on said input signal, on the ba- 
sis of said psychological auditory sense encod- 
ed information. 

40 

1 2. The program storing medium according to claim 1 1 , 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- 
45 fomnation is generated, on the basis of the threshold 
of audibility. 

1 3. The program storing medium according to claim 1 1 , 

wherein 

50 at said psychological auditory sense analysis 

step, said psychological auditory sense encoded in- 
formation is generated, on the basis of the masking 
effect of the audio signal. 

55 1 4. The program storing medium according to claim 1 1 , 
wherein 

at said step of superimposing said digital wa- 
termarking information on said input signal, digital 
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watermarking Information that has large compres- 
sion resistance toward said compression process- 
ing and digital watermarking information that has 
small compression resistance toward said com- 
pression processing are superimposed on said in- s 
put signal, on the basis of said psychological audi- 
tory sense encoded Information. 

15. The program storing medium according to claim 1 1 , 
wherein io 

said digital watemnarking infonnatlon Is the 
copyright information relevant to said input signal. 
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